Repair voice turns that claim a knowledge gap without searching - #20
Open
codeyogi911 wants to merge 1 commit into
Open
Repair voice turns that claim a knowledge gap without searching#20codeyogi911 wants to merge 1 commit into
codeyogi911 wants to merge 1 commit into
Conversation
The voice model at temperature 0 skips search_help_center on unfamiliar-product questions and answers "I don't have information on that" directly, violating the prompt's search-before-claiming rule and flaking the kb_no_match_honest eval bar (observed 7/8 runs). The broken shape is deterministic to detect — no tool call plus a missing-documentation claim — so such a turn is re-run once with the search forced on the first step via prepareStep, and only the repaired turn reaches the caller. Off-topic scope redirects phrase refusals, not knowledge gaps, so they never repair and never gain a tool call, keeping the sibling support_scope bar intact. Wired into the production stream in demo-agent and both eval-worker paths. Detector regexes are corpus-driven from probed model output; voice-repair.test.ts pins both reply classes. Verified with 5 consecutive green eval:voice runs and npm run check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The voice eval bar "how-to question must search the help centre before answering" (kb_no_match_honest) was model-flaky:
@cf/zai-org/glm-4.7-flashat temperature 0 answers unfamiliar-product questions with "I don't have information on that" without callingsearch_help_center, despite explicit prompt instructions. Probing showed the flake at 7 of 8 runs. The previously reverted fix (forcingtoolChoiceon question-shaped first steps) broke the sibling bar "off-topic question must not call a support tool" because it classified the question; off-topic questions are also question-shaped.Approach
Classify the reply instead. Probing 30+ live turns showed the two failure classes are lexically separable:
New
src/voice/repair.tsenforces the invariant the prompt already states (never claim something is undocumented without a search this turn):claimsKnowledgeGap(text)— corpus-driven detector for missing-documentation claims, with guards so refusals, bare "no answer" troubleshooting, and "general questions" redirects never match.repairUnsearchedKnowledgeGap(stream, rerun)— buffers the turn until the first tool call (tool-using turns pass through with zero added latency); a tool-less turn ending in a gap claim is discarded and re-run once withsearch_help_centerforced on step 0 viaprepareStep. The rerun always searches, so it can never re-trigger; ticket creation is already requestId-idempotent.Wired into the production stream in
demo-agent.tsand mirrored in both eval-worker paths (generateTextandstreamText).Verification
npm run eval:voiceruns (22 passes each)npm run checkgreen (234 + 6 tests, all typechecks, generated assets clean)test/voice-repair.test.tspins the detector to the observed reply corpus for both classes plus the wrapper's pass-through/disarm/repair behaviorsKnown residual (pre-existing, out of scope): 1 of 6 probed kb_grounded_answer runs invented an undocumented interval ("monthly"); it never tripped across the 5 eval runs and this repair intentionally does not cover invention.
🤖 Generated with Claude Code